{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15行代码搞定Kaggle电影评论情感分析\n",
    "\n",
    ">同济大学 张子豪 2019-6-28\n",
    "\n",
    ">Bilibili视频：同济子豪兄\n",
    "\n",
    "接下来的20分钟，我将手把手带你参加Kaggle数据科学竞赛：电影评论情感分析与文本数据挖掘，用不到20行代码超过百分之五十的参赛选手。<br>\n",
    "你将掌握文本数据预处理、去除停用词、词袋模型、TF-IDF模型等自然语言处理和文本数据挖掘的基础知识，并掌握数据集拆分、逻辑回归模型、超参数的网格搜索、交叉验证、模型效果评价等机器学习基础知识，了解参加Kaggle数据竞赛的流程。\n",
    "\n",
    "通过本案例，我想告诉你三件事：\n",
    "\n",
    "- Kaggle数据竞赛没什么可怕的，大部分的参赛选手都比较水。\n",
    "- 黑猫白猫，抓到老鼠就是好猫。算法不是越复杂越好，有时简单的算法效果也很不错，甚至“一招逻辑回归包打天下”。\n",
    "- 数据预处理和特征工程很重要，对于文本数据来说，有词袋模型、TF-IDF模型、word2vec模型等方法来抽取文本的特征。\n",
    "\n",
    "如果你不太会使用现在看到的这个jupyter notebook工具，请看子豪兄的这个视频\n",
    "\n",
    "[python数据分析神器Jupyter notebook快速入门](https://www.bilibili.com/video/av54100790)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据集介绍\n",
    "\n",
    "Kaggle竞赛网址：\n",
    "https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews\n",
    "\n",
    "![数据集介绍](https://upload-images.jianshu.io/upload_images/13714448-a8acedd302258ce7.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\n",
    "\n",
    "![排行榜](https://upload-images.jianshu.io/upload_images/13714448-3955c5fc364d919d.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\n",
    "\n",
    "\n",
    "![参赛队与一些公开的kernel](https://upload-images.jianshu.io/upload_images/13714448-21af581b730bf01c.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 下载所需工具包\n",
    "\n",
    "在命令行界中运行下面这行命令，从清华大学开源软件镜像站下载pandas、sklearn这两个python第三方模块。\n",
    "\n",
    "`pip install pandas sklearn -i https://pypi.tuna.tsinghua.edu.cn/simple`\n",
    "\n",
    "如果命令行提示找不到pip命令，那说明你没有安装Python，请看子豪兄的视频\n",
    "[一劳永逸安装和配置python3.7.2（新手必读）](https://www.bilibili.com/video/av45546100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 导入数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pandas是用来导入、整理、清洗表格数据的专用工具，类似excel，但功能更加强大，导入的时候给pandas起个小名叫pd\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用pandas的read_csv函数读取训练数据及测试数据，数据文件是.tsv格式的，也就是说数据用制表符\\t分隔，类似于.csv文件的数据用逗号分隔\n",
    "data_train = pd.read_csv('./train.tsv',sep='\\t')\n",
    "data_test = pd.read_csv('./test.tsv',sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>SentenceId</th>\n",
       "      <th>Phrase</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>A series of escapades demonstrating the adage ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>A series of escapades demonstrating the adage ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>A series</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>series</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PhraseId  SentenceId                                             Phrase  \\\n",
       "0         1           1  A series of escapades demonstrating the adage ...   \n",
       "1         2           1  A series of escapades demonstrating the adage ...   \n",
       "2         3           1                                           A series   \n",
       "3         4           1                                                  A   \n",
       "4         5           1                                             series   \n",
       "\n",
       "   Sentiment  \n",
       "0          1  \n",
       "1          2  \n",
       "2          2  \n",
       "3          2  \n",
       "4          2  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 看训练集数据前5行，Phrase列为电影评论文本，Sentiment为情感标签\n",
    "data_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "电影评论的情感标签\n",
    "\n",
    "- 0 - negative\n",
    "- 1 - somewhat negative\n",
    "- 2 - neutral\n",
    "- 3 - somewhat positive\n",
    "- 4 - positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(156060, 4)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 共有156060行训练数据，每行数据都有短语ID、句子ID、文本内容、情感标签四列\n",
    "data_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>SentenceId</th>\n",
       "      <th>Phrase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>156061</td>\n",
       "      <td>8545</td>\n",
       "      <td>An intermittently pleasing but mostly routine ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>156062</td>\n",
       "      <td>8545</td>\n",
       "      <td>An intermittently pleasing but mostly routine ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>156063</td>\n",
       "      <td>8545</td>\n",
       "      <td>An</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>156064</td>\n",
       "      <td>8545</td>\n",
       "      <td>intermittently pleasing but mostly routine effort</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>156065</td>\n",
       "      <td>8545</td>\n",
       "      <td>intermittently pleasing but mostly routine</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PhraseId  SentenceId                                             Phrase\n",
       "0    156061        8545  An intermittently pleasing but mostly routine ...\n",
       "1    156062        8545  An intermittently pleasing but mostly routine ...\n",
       "2    156063        8545                                                 An\n",
       "3    156064        8545  intermittently pleasing but mostly routine effort\n",
       "4    156065        8545         intermittently pleasing but mostly routine"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看测试集数据前5行，Phrase列就是需要我们自己构建模型预测情感标签的文本\n",
    "data_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(66292, 3)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 共有66292行测试集数据，每个数据都有短语ID、句子ID、文本内容三列\n",
    "data_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 构建语料库"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 我们需要对文本进行一些处理，将原始文本中的每一个词变成计算机看得懂的向量，这一过程叫做文本的特征工程，非常重要。\n",
    "- 有很多将词变成向量的方法，比如下面将要介绍的词袋模型、TF-IDF模型，以及视频中介绍的word2vec模型。\n",
    "- 不管采用什么模型，我们都需要先把训练集和测试集中所有文本内容组合在一起，构建一个语料库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 提取训练集中的文本内容 \n",
    "train_sentences = data_train['Phrase']\n",
    "\n",
    "# 提取测试集中的文本内容\n",
    "test_sentences = data_test['Phrase']\n",
    "\n",
    "# 通过pandas的concat函数将训练集和测试集的文本内容合并到一起\n",
    "sentences = pd.concat([train_sentences,test_sentences])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(222352,)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 合并到一起的语料库共有222352行数据\n",
    "sentences.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 提取训练集中的情感标签，一共是156060个标签\n",
    "label = data_train['Sentiment']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(156060,)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入停词库，停词库中的词是一些废话单词和语气词，对情感分析没什么帮助\n",
    "stop_words = open('./stop_words.txt',encoding='utf-8').read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"\\ufeffain'\",\n",
       " 'happy',\n",
       " 'isn',\n",
       " 'ain',\n",
       " 'al',\n",
       " 'couldn',\n",
       " 'didn',\n",
       " 'doesn',\n",
       " 'hadn',\n",
       " 'hasn',\n",
       " 'haven',\n",
       " 'sn',\n",
       " 'll',\n",
       " 'mon',\n",
       " 'shouldn',\n",
       " 've',\n",
       " 'wasn',\n",
       " 'weren',\n",
       " 'won',\n",
       " 'wouldn',\n",
       " \"'d\",\n",
       " \"'ll\",\n",
       " \"'m\",\n",
       " \"'re\",\n",
       " \"'s\",\n",
       " \"'t\",\n",
       " \"'ve\",\n",
       " 'ZT',\n",
       " 'ZZ',\n",
       " 'a',\n",
       " \"a's\",\n",
       " 'able',\n",
       " 'about',\n",
       " 'above',\n",
       " 'abst',\n",
       " 'accordance',\n",
       " 'according',\n",
       " 'accordingly',\n",
       " 'across',\n",
       " 'act',\n",
       " 'actually',\n",
       " 'added',\n",
       " 'adj',\n",
       " 'adopted',\n",
       " 'affected',\n",
       " 'affecting',\n",
       " 'affects',\n",
       " 'after',\n",
       " 'afterwards',\n",
       " 'again',\n",
       " 'against',\n",
       " 'ah',\n",
       " \"ain't\",\n",
       " 'all',\n",
       " 'allow',\n",
       " 'allows',\n",
       " 'almost',\n",
       " 'alone',\n",
       " 'along',\n",
       " 'already',\n",
       " 'also',\n",
       " 'although',\n",
       " 'always',\n",
       " 'am',\n",
       " 'among',\n",
       " 'amongst',\n",
       " 'an',\n",
       " 'and',\n",
       " 'announce',\n",
       " 'another',\n",
       " 'any',\n",
       " 'anybody',\n",
       " 'anyhow',\n",
       " 'anymore',\n",
       " 'anyone',\n",
       " 'anything',\n",
       " 'anyway',\n",
       " 'anyways',\n",
       " 'anywhere',\n",
       " 'apart',\n",
       " 'apparently',\n",
       " 'appear',\n",
       " 'appreciate',\n",
       " 'appropriate',\n",
       " 'approximately',\n",
       " 'are',\n",
       " 'area',\n",
       " 'areas',\n",
       " 'aren',\n",
       " \"aren't\",\n",
       " 'arent',\n",
       " 'arise',\n",
       " 'around',\n",
       " 'as',\n",
       " 'aside',\n",
       " 'ask',\n",
       " 'asked',\n",
       " 'asking',\n",
       " 'asks',\n",
       " 'associated',\n",
       " 'at',\n",
       " 'auth',\n",
       " 'available',\n",
       " 'away',\n",
       " 'awfully',\n",
       " 'b',\n",
       " 'back',\n",
       " 'backed',\n",
       " 'backing',\n",
       " 'backs',\n",
       " 'be',\n",
       " 'became',\n",
       " 'because',\n",
       " 'become',\n",
       " 'becomes',\n",
       " 'becoming',\n",
       " 'been',\n",
       " 'before',\n",
       " 'beforehand',\n",
       " 'began',\n",
       " 'begin',\n",
       " 'beginning',\n",
       " 'beginnings',\n",
       " 'begins',\n",
       " 'behind',\n",
       " 'being',\n",
       " 'beings',\n",
       " 'believe',\n",
       " 'below',\n",
       " 'beside',\n",
       " 'besides',\n",
       " 'best',\n",
       " 'better',\n",
       " 'between',\n",
       " 'beyond',\n",
       " 'big',\n",
       " 'biol',\n",
       " 'both',\n",
       " 'brief',\n",
       " 'briefly',\n",
       " 'but',\n",
       " 'by',\n",
       " 'c',\n",
       " \"c'mon\",\n",
       " \"c's\",\n",
       " 'ca',\n",
       " 'came',\n",
       " 'can',\n",
       " \"can't\",\n",
       " 'cannot',\n",
       " 'cant',\n",
       " 'case',\n",
       " 'cases',\n",
       " 'cause',\n",
       " 'causes',\n",
       " 'certain',\n",
       " 'certainly',\n",
       " 'changes',\n",
       " 'clear',\n",
       " 'clearly',\n",
       " 'co',\n",
       " 'com',\n",
       " 'come',\n",
       " 'comes',\n",
       " 'concerning',\n",
       " 'consequently',\n",
       " 'consider',\n",
       " 'considering',\n",
       " 'contain',\n",
       " 'containing',\n",
       " 'contains',\n",
       " 'corresponding',\n",
       " 'could',\n",
       " \"couldn't\",\n",
       " 'couldnt',\n",
       " 'course',\n",
       " 'currently',\n",
       " 'd',\n",
       " 'date',\n",
       " 'definitely',\n",
       " 'describe',\n",
       " 'described',\n",
       " 'despite',\n",
       " 'did',\n",
       " \"didn't\",\n",
       " 'differ',\n",
       " 'different',\n",
       " 'differently',\n",
       " 'discuss',\n",
       " 'do',\n",
       " 'does',\n",
       " \"doesn't\",\n",
       " 'doing',\n",
       " \"don't\",\n",
       " 'done',\n",
       " 'down',\n",
       " 'downed',\n",
       " 'downing',\n",
       " 'downs',\n",
       " 'downwards',\n",
       " 'due',\n",
       " 'during',\n",
       " 'e',\n",
       " 'each',\n",
       " 'early',\n",
       " 'ed',\n",
       " 'edu',\n",
       " 'effect',\n",
       " 'eg',\n",
       " 'eight',\n",
       " 'eighty',\n",
       " 'either',\n",
       " 'else',\n",
       " 'elsewhere',\n",
       " 'end',\n",
       " 'ended',\n",
       " 'ending',\n",
       " 'ends',\n",
       " 'enough',\n",
       " 'entirely',\n",
       " 'especially',\n",
       " 'et',\n",
       " 'et-al',\n",
       " 'etc',\n",
       " 'even',\n",
       " 'evenly',\n",
       " 'ever',\n",
       " 'every',\n",
       " 'everybody',\n",
       " 'everyone',\n",
       " 'everything',\n",
       " 'everywhere',\n",
       " 'ex',\n",
       " 'exactly',\n",
       " 'example',\n",
       " 'except',\n",
       " 'f',\n",
       " 'face',\n",
       " 'faces',\n",
       " 'fact',\n",
       " 'facts',\n",
       " 'far',\n",
       " 'felt',\n",
       " 'few',\n",
       " 'ff',\n",
       " 'fifth',\n",
       " 'find',\n",
       " 'finds',\n",
       " 'first',\n",
       " 'five',\n",
       " 'fix',\n",
       " 'followed',\n",
       " 'following',\n",
       " 'follows',\n",
       " 'for',\n",
       " 'former',\n",
       " 'formerly',\n",
       " 'forth',\n",
       " 'found',\n",
       " 'four',\n",
       " 'from',\n",
       " 'full',\n",
       " 'fully',\n",
       " 'further',\n",
       " 'furthered',\n",
       " 'furthering',\n",
       " 'furthermore',\n",
       " 'furthers',\n",
       " 'g',\n",
       " 'gave',\n",
       " 'general',\n",
       " 'generally',\n",
       " 'get',\n",
       " 'gets',\n",
       " 'getting',\n",
       " 'give',\n",
       " 'given',\n",
       " 'gives',\n",
       " 'giving',\n",
       " 'go',\n",
       " 'goes',\n",
       " 'going',\n",
       " 'gone',\n",
       " 'good',\n",
       " 'goods',\n",
       " 'got',\n",
       " 'gotten',\n",
       " 'great',\n",
       " 'greater',\n",
       " 'greatest',\n",
       " 'greetings',\n",
       " 'group',\n",
       " 'grouped',\n",
       " 'grouping',\n",
       " 'groups',\n",
       " 'h',\n",
       " 'had',\n",
       " \"hadn't\",\n",
       " 'happens',\n",
       " 'hardly',\n",
       " 'has',\n",
       " \"hasn't\",\n",
       " 'have',\n",
       " \"haven't\",\n",
       " 'having',\n",
       " 'he',\n",
       " \"he's\",\n",
       " 'hed',\n",
       " 'hello',\n",
       " 'help',\n",
       " 'hence',\n",
       " 'her',\n",
       " 'here',\n",
       " \"here's\",\n",
       " 'hereafter',\n",
       " 'hereby',\n",
       " 'herein',\n",
       " 'heres',\n",
       " 'hereupon',\n",
       " 'hers',\n",
       " 'herself',\n",
       " 'hes',\n",
       " 'hi',\n",
       " 'hid',\n",
       " 'high',\n",
       " 'higher',\n",
       " 'highest',\n",
       " 'him',\n",
       " 'himself',\n",
       " 'his',\n",
       " 'hither',\n",
       " 'home',\n",
       " 'hopefully',\n",
       " 'how',\n",
       " 'howbeit',\n",
       " 'however',\n",
       " 'hundred',\n",
       " 'i',\n",
       " \"i'd\",\n",
       " \"i'll\",\n",
       " \"i'm\",\n",
       " \"i've\",\n",
       " 'id',\n",
       " 'ie',\n",
       " 'if',\n",
       " 'ignored',\n",
       " 'im',\n",
       " 'immediate',\n",
       " 'immediately',\n",
       " 'importance',\n",
       " 'important',\n",
       " 'in',\n",
       " 'inasmuch',\n",
       " 'inc',\n",
       " 'include',\n",
       " 'indeed',\n",
       " 'index',\n",
       " 'indicate',\n",
       " 'indicated',\n",
       " 'indicates',\n",
       " 'information',\n",
       " 'inner',\n",
       " 'insofar',\n",
       " 'instead',\n",
       " 'interest',\n",
       " 'interested',\n",
       " 'interesting',\n",
       " 'interests',\n",
       " 'into',\n",
       " 'invention',\n",
       " 'inward',\n",
       " 'is',\n",
       " \"isn't\",\n",
       " 'it',\n",
       " \"it'd\",\n",
       " \"it'll\",\n",
       " \"it's\",\n",
       " 'itd',\n",
       " 'its',\n",
       " 'itself',\n",
       " 'j',\n",
       " 'just',\n",
       " 'k',\n",
       " 'keep',\n",
       " 'keeps',\n",
       " 'kept',\n",
       " 'keys',\n",
       " 'kg',\n",
       " 'kind',\n",
       " 'km',\n",
       " 'knew',\n",
       " 'know',\n",
       " 'known',\n",
       " 'knows',\n",
       " 'l',\n",
       " 'large',\n",
       " 'largely',\n",
       " 'last',\n",
       " 'lately',\n",
       " 'later',\n",
       " 'latest',\n",
       " 'latter',\n",
       " 'latterly',\n",
       " 'least',\n",
       " 'less',\n",
       " 'lest',\n",
       " 'let',\n",
       " \"let's\",\n",
       " 'lets',\n",
       " 'like',\n",
       " 'liked',\n",
       " 'likely',\n",
       " 'line',\n",
       " 'little',\n",
       " 'long',\n",
       " 'longer',\n",
       " 'longest',\n",
       " 'look',\n",
       " 'looking',\n",
       " 'looks',\n",
       " 'ltd',\n",
       " 'm',\n",
       " 'made',\n",
       " 'mainly',\n",
       " 'make',\n",
       " 'makes',\n",
       " 'making',\n",
       " 'man',\n",
       " 'many',\n",
       " 'may',\n",
       " 'maybe',\n",
       " 'me',\n",
       " 'mean',\n",
       " 'means',\n",
       " 'meantime',\n",
       " 'meanwhile',\n",
       " 'member',\n",
       " 'members',\n",
       " 'men',\n",
       " 'merely',\n",
       " 'mg',\n",
       " 'might',\n",
       " 'million',\n",
       " 'miss',\n",
       " 'ml',\n",
       " 'more',\n",
       " 'moreover',\n",
       " 'most',\n",
       " 'mostly',\n",
       " 'mr',\n",
       " 'mrs',\n",
       " 'much',\n",
       " 'mug',\n",
       " 'must',\n",
       " 'my',\n",
       " 'myself',\n",
       " 'n',\n",
       " \"n't\",\n",
       " 'na',\n",
       " 'name',\n",
       " 'namely',\n",
       " 'nay',\n",
       " 'nd',\n",
       " 'near',\n",
       " 'nearly',\n",
       " 'necessarily',\n",
       " 'necessary',\n",
       " 'need',\n",
       " 'needed',\n",
       " 'needing',\n",
       " 'needs',\n",
       " 'neither',\n",
       " 'never',\n",
       " 'nevertheless',\n",
       " 'new',\n",
       " 'newer',\n",
       " 'newest',\n",
       " 'next',\n",
       " 'nine',\n",
       " 'ninety',\n",
       " 'no',\n",
       " 'nobody',\n",
       " 'non',\n",
       " 'none',\n",
       " 'nonetheless',\n",
       " 'noone',\n",
       " 'nor',\n",
       " 'normally',\n",
       " 'nos',\n",
       " 'not',\n",
       " 'noted',\n",
       " 'nothing',\n",
       " 'novel',\n",
       " 'now',\n",
       " 'nowhere',\n",
       " 'number',\n",
       " 'numbers',\n",
       " 'o',\n",
       " 'obtain',\n",
       " 'obtained',\n",
       " 'obviously',\n",
       " 'of',\n",
       " 'off',\n",
       " 'often',\n",
       " 'oh',\n",
       " 'ok',\n",
       " 'okay',\n",
       " 'old',\n",
       " 'older',\n",
       " 'oldest',\n",
       " 'omitted',\n",
       " 'on',\n",
       " 'once',\n",
       " 'one',\n",
       " 'ones',\n",
       " 'only',\n",
       " 'onto',\n",
       " 'open',\n",
       " 'opened',\n",
       " 'opening',\n",
       " 'opens',\n",
       " 'or',\n",
       " 'ord',\n",
       " 'order',\n",
       " 'ordered',\n",
       " 'ordering',\n",
       " 'orders',\n",
       " 'other',\n",
       " 'others',\n",
       " 'otherwise',\n",
       " 'ought',\n",
       " 'our',\n",
       " 'ours',\n",
       " 'ourselves',\n",
       " 'out',\n",
       " 'outside',\n",
       " 'over',\n",
       " 'overall',\n",
       " 'owing',\n",
       " 'own',\n",
       " 'p',\n",
       " 'page',\n",
       " 'pages',\n",
       " 'part',\n",
       " 'parted',\n",
       " 'particular',\n",
       " 'particularly',\n",
       " 'parting',\n",
       " 'parts',\n",
       " 'past',\n",
       " 'per',\n",
       " 'perhaps',\n",
       " 'place',\n",
       " 'placed',\n",
       " 'places',\n",
       " 'please',\n",
       " 'plus',\n",
       " 'point',\n",
       " 'pointed',\n",
       " 'pointing',\n",
       " 'points',\n",
       " 'poorly',\n",
       " 'possible',\n",
       " 'possibly',\n",
       " 'potentially',\n",
       " 'pp',\n",
       " 'predominantly',\n",
       " 'present',\n",
       " 'presented',\n",
       " 'presenting',\n",
       " 'presents',\n",
       " 'presumably',\n",
       " 'previously',\n",
       " 'primarily',\n",
       " 'probably',\n",
       " 'problem',\n",
       " 'problems',\n",
       " 'promptly',\n",
       " 'proud',\n",
       " 'provides',\n",
       " 'put',\n",
       " 'puts',\n",
       " 'q',\n",
       " 'que',\n",
       " 'quickly',\n",
       " 'quite',\n",
       " 'qv',\n",
       " 'r',\n",
       " 'ran',\n",
       " 'rather',\n",
       " 'rd',\n",
       " 're',\n",
       " 'readily',\n",
       " 'really',\n",
       " 'reasonably',\n",
       " 'recent',\n",
       " 'recently',\n",
       " 'ref',\n",
       " 'refs',\n",
       " 'regarding',\n",
       " 'regardless',\n",
       " 'regards',\n",
       " 'related',\n",
       " 'relatively',\n",
       " 'research',\n",
       " 'respectively',\n",
       " 'resulted',\n",
       " 'resulting',\n",
       " 'results',\n",
       " 'right',\n",
       " 'room',\n",
       " 'rooms',\n",
       " 'run',\n",
       " 's',\n",
       " 'said',\n",
       " 'same',\n",
       " 'saw',\n",
       " 'say',\n",
       " 'saying',\n",
       " 'says',\n",
       " 'sec',\n",
       " 'second',\n",
       " 'secondly',\n",
       " 'seconds',\n",
       " 'section',\n",
       " 'see',\n",
       " 'seeing',\n",
       " 'seem',\n",
       " 'seemed',\n",
       " 'seeming',\n",
       " 'seems',\n",
       " 'seen',\n",
       " 'sees',\n",
       " 'self',\n",
       " 'selves',\n",
       " 'sensible',\n",
       " 'sent',\n",
       " 'serious',\n",
       " 'seriously',\n",
       " 'seven',\n",
       " 'several',\n",
       " 'shall',\n",
       " 'she',\n",
       " \"she'll\",\n",
       " 'shed',\n",
       " 'shes',\n",
       " 'should',\n",
       " \"shouldn't\",\n",
       " 'show',\n",
       " 'showed',\n",
       " 'showing',\n",
       " 'shown',\n",
       " 'showns',\n",
       " 'shows',\n",
       " 'side',\n",
       " 'sides',\n",
       " 'significant',\n",
       " 'significantly',\n",
       " 'similar',\n",
       " 'similarly',\n",
       " 'since',\n",
       " 'six',\n",
       " 'slightly',\n",
       " 'small',\n",
       " 'smaller',\n",
       " 'smallest',\n",
       " 'so',\n",
       " 'some',\n",
       " 'somebody',\n",
       " 'somehow',\n",
       " 'someone',\n",
       " 'somethan',\n",
       " 'something',\n",
       " 'sometime',\n",
       " 'sometimes',\n",
       " 'somewhat',\n",
       " 'somewhere',\n",
       " 'soon',\n",
       " 'sorry',\n",
       " 'specifically',\n",
       " 'specified',\n",
       " 'specify',\n",
       " 'specifying',\n",
       " 'state',\n",
       " 'states',\n",
       " 'still',\n",
       " 'stop',\n",
       " 'strongly',\n",
       " 'sub',\n",
       " 'substantially',\n",
       " 'successfully',\n",
       " 'such',\n",
       " 'sufficiently',\n",
       " 'suggest',\n",
       " 'sup',\n",
       " 'sure',\n",
       " 't',\n",
       " \"t's\",\n",
       " 'take',\n",
       " 'taken',\n",
       " 'taking',\n",
       " 'tell',\n",
       " 'tends',\n",
       " 'th',\n",
       " 'than',\n",
       " 'thank',\n",
       " 'thanks',\n",
       " 'thanx',\n",
       " 'that',\n",
       " \"that'll\",\n",
       " \"that's\",\n",
       " \"that've\",\n",
       " 'thats',\n",
       " 'the',\n",
       " 'their',\n",
       " 'theirs',\n",
       " 'them',\n",
       " 'themselves',\n",
       " 'then',\n",
       " 'thence',\n",
       " 'there',\n",
       " \"there'll\",\n",
       " \"there's\",\n",
       " \"there've\",\n",
       " 'thereafter',\n",
       " 'thereby',\n",
       " 'thered',\n",
       " 'therefore',\n",
       " 'therein',\n",
       " 'thereof',\n",
       " 'therere',\n",
       " 'theres',\n",
       " 'thereto',\n",
       " 'thereupon',\n",
       " 'these',\n",
       " 'they',\n",
       " \"they'd\",\n",
       " \"they'll\",\n",
       " \"they're\",\n",
       " \"they've\",\n",
       " 'theyd',\n",
       " 'theyre',\n",
       " 'thing',\n",
       " 'things',\n",
       " 'think',\n",
       " 'thinks',\n",
       " 'third',\n",
       " 'this',\n",
       " 'thorough',\n",
       " 'thoroughly',\n",
       " 'those',\n",
       " 'thou',\n",
       " 'though',\n",
       " 'thoughh',\n",
       " 'thought',\n",
       " 'thoughts',\n",
       " 'thousand',\n",
       " 'three',\n",
       " 'throug',\n",
       " 'through',\n",
       " 'throughout',\n",
       " 'thru',\n",
       " 'thus',\n",
       " 'til',\n",
       " 'tip',\n",
       " 'to',\n",
       " 'today',\n",
       " 'together',\n",
       " 'too',\n",
       " 'took',\n",
       " 'toward',\n",
       " 'towards',\n",
       " 'tried',\n",
       " 'tries',\n",
       " 'truly',\n",
       " 'try',\n",
       " 'trying',\n",
       " 'ts',\n",
       " 'turn',\n",
       " 'turned',\n",
       " 'turning',\n",
       " 'turns',\n",
       " 'twice',\n",
       " 'two',\n",
       " 'u',\n",
       " 'un',\n",
       " 'under',\n",
       " 'unfortunately',\n",
       " 'unless',\n",
       " 'unlike',\n",
       " 'unlikely',\n",
       " 'until',\n",
       " 'unto',\n",
       " 'up',\n",
       " 'upon',\n",
       " 'ups',\n",
       " 'us',\n",
       " 'use',\n",
       " 'used',\n",
       " 'useful',\n",
       " 'usefully',\n",
       " 'usefulness',\n",
       " 'uses',\n",
       " 'using',\n",
       " 'usually',\n",
       " 'uucp',\n",
       " 'v',\n",
       " 'value',\n",
       " 'various',\n",
       " 'very',\n",
       " 'via',\n",
       " 'viz',\n",
       " 'vol',\n",
       " 'vols',\n",
       " 'vs',\n",
       " 'w',\n",
       " 'want',\n",
       " 'wanted',\n",
       " 'wanting',\n",
       " 'wants',\n",
       " 'was',\n",
       " \"wasn't\",\n",
       " 'way',\n",
       " 'ways',\n",
       " 'we',\n",
       " \"we'd\",\n",
       " \"we'll\",\n",
       " \"we're\",\n",
       " \"we've\",\n",
       " 'wed',\n",
       " 'welcome',\n",
       " 'well',\n",
       " 'wells',\n",
       " 'went',\n",
       " 'were',\n",
       " \"weren't\",\n",
       " 'what',\n",
       " \"what'll\",\n",
       " \"what's\",\n",
       " 'whatever',\n",
       " 'whats',\n",
       " 'when',\n",
       " 'whence',\n",
       " 'whenever',\n",
       " 'where',\n",
       " \"where's\",\n",
       " 'whereafter',\n",
       " 'whereas',\n",
       " 'whereby',\n",
       " 'wherein',\n",
       " 'wheres',\n",
       " 'whereupon',\n",
       " 'wherever',\n",
       " 'whether',\n",
       " 'which',\n",
       " 'while',\n",
       " 'whim',\n",
       " 'whither',\n",
       " 'who',\n",
       " \"who'll\",\n",
       " \"who's\",\n",
       " 'whod',\n",
       " 'whoever',\n",
       " 'whole',\n",
       " 'whom',\n",
       " 'whomever',\n",
       " 'whos',\n",
       " 'whose',\n",
       " 'why',\n",
       " 'widely',\n",
       " 'will',\n",
       " 'willing',\n",
       " 'wish',\n",
       " 'with',\n",
       " 'within',\n",
       " 'without',\n",
       " \"won't\",\n",
       " 'wonder',\n",
       " 'words',\n",
       " 'work',\n",
       " 'worked',\n",
       " 'working',\n",
       " 'works',\n",
       " 'world',\n",
       " 'would',\n",
       " \"wouldn't\",\n",
       " 'www',\n",
       " 'x',\n",
       " 'y',\n",
       " 'year',\n",
       " 'years',\n",
       " 'yes',\n",
       " 'yet',\n",
       " 'you',\n",
       " \"you'd\",\n",
       " \"you'll\",\n",
       " \"you're\",\n",
       " \"you've\",\n",
       " 'youd',\n",
       " 'young',\n",
       " 'younger',\n",
       " 'youngest',\n",
       " 'your',\n",
       " 'youre',\n",
       " 'yours',\n",
       " 'yourself',\n",
       " 'yourselves',\n",
       " 'z',\n",
       " 'zero',\n",
       " 'zt',\n",
       " 'zz',\n",
       " '',\n",
       " 'not',\n",
       " 'also',\n",
       " 'EVEN',\n",
       " 'They',\n",
       " 'some',\n",
       " 'don',\n",
       " 'what',\n",
       " 'Of',\n",
       " 'how',\n",
       " 'Happy',\n",
       " 'Yes',\n",
       " 'YES',\n",
       " 'yes',\n",
       " 'No',\n",
       " 'NO',\n",
       " 'no',\n",
       " 'It',\n",
       " 'Its',\n",
       " 'etc',\n",
       " 'most',\n",
       " 'For',\n",
       " 'lulu',\n",
       " 'Is',\n",
       " 'is',\n",
       " 'do',\n",
       " 'Do',\n",
       " 'can',\n",
       " 'be',\n",
       " 'are',\n",
       " 'by',\n",
       " 'You',\n",
       " 'an',\n",
       " 'The',\n",
       " 'www',\n",
       " 'at',\n",
       " 'any',\n",
       " 'one',\n",
       " 'our',\n",
       " 'null',\n",
       " 'com',\n",
       " 'str',\n",
       " 'and',\n",
       " 'to',\n",
       " 'for',\n",
       " 'the',\n",
       " 'you',\n",
       " 'is',\n",
       " 'of',\n",
       " 'in',\n",
       " 'on',\n",
       " 'can',\n",
       " 'or',\n",
       " 'de',\n",
       " 'as',\n",
       " 'shtml',\n",
       " 'your',\n",
       " 'time',\n",
       " 'like',\n",
       " 'other',\n",
       " 'make',\n",
       " 'Look',\n",
       " 'with',\n",
       " 'from',\n",
       " 'this',\n",
       " 'will',\n",
       " 'that',\n",
       " 'more',\n",
       " 'This',\n",
       " 'http',\n",
       " 'have',\n",
       " 'https',\n",
       " ',',\n",
       " '.',\n",
       " '<',\n",
       " '>',\n",
       " '?',\n",
       " '/',\n",
       " '\\\\',\n",
       " '|',\n",
       " '-',\n",
       " '_',\n",
       " '+',\n",
       " '=',\n",
       " '*',\n",
       " '&',\n",
       " '^',\n",
       " '%',\n",
       " '#',\n",
       " ...]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# stop_words是一个列表，列表中每一个元素都是一个停用词\n",
    "stop_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用词袋模型进行文本特征工程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![词袋模型](https://upload-images.jianshu.io/upload_images/13714448-63922809a80864ff.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用sklearn库中的CountVectorizer构建词袋模型\n",
    "# 词袋模型的详细介绍请看子豪兄的视频\n",
    "# analyzer='word'指的是以词为单位进行分析，对于拉丁语系语言，有时需要以字母'character'为单位进行分析\n",
    "# ngram指分析相邻的几个词，避免原始的词袋模型中词序丢失的问题\n",
    "# max_features指最终的词袋矩阵里面包含语料库中出现次数最多的多少个词\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "co = CountVectorizer(\n",
    "    analyzer='word',\n",
    "    ngram_range=(1,4),\n",
    "    stop_words=stop_words,\n",
    "    max_features=150000\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=150000, min_df=1,\n",
       "        ngram_range=(1, 4), preprocessor=None,\n",
       "        stop_words=[\"\\ufeffain'\", 'happy', 'isn', 'ain', 'al', 'couldn', 'didn', 'doesn', 'hadn', 'hasn', 'haven', 'sn', 'll', 'mon', 'shouldn', 've', 'wasn', 'weren', 'won', 'wouldn', \"'d\", \"'ll\", \"'m\", \"'re\", \"'s\", \"'t\", \"'ve\", 'ZT', 'ZZ', 'a', \"a's\", 'able', 'about', 'above', 'abst', 'accordance', 'accor...', ',', '·', '￥', '……', '（', '）', '——', '、', '：', '；', '“', '’', '《', '》', '，', '。', '、', '？', '★ '],\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用语料库，构建词袋模型\n",
    "\n",
    "co.fit(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将训练集随机拆分为新的训练集和验证集，默认3:1,然后进行词频统计\n",
    "# 在机器学习中，训练集相当于课后习题，用于平时学习知识。验证集相当于模拟考试，用于检验学习成果。测试集相当于高考，用于最终Kaggle竞赛打分。\n",
    "# 新的训练集和验证集都来自于最初的训练集，都是有标签的。\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "x_train,x_test,y_train,y_test = train_test_split(train_sentences,label,random_state=1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- x_train 训练集数据 （相当于课后习题）\n",
    "- x_test 验证集数据 （相当于模拟考试题）\n",
    "- y_train 训练集标签 （相当于课后习题答案）\n",
    "- y_test 验证集标签（相当于模拟考试题答案）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A series of escapades demonstrating the adage that what is good for the goose'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 随便看训练集中的一个数据\n",
    "x_train[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用上面构建的词袋模型，把训练集和验证集中的每一个词都进行特征工程，变成向量\n",
    "\n",
    "x_train = co.transform(x_train)\n",
    "x_test = co.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x150000 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 6 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 随便看训练集中的一个数据，它是150000列的稀疏矩阵\n",
    "\n",
    "x_train[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 构建分类器算法，对词袋模型处理后的文本进行机器学习和数据挖掘"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 逻辑回归分类器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![逻辑回归](https://upload-images.jianshu.io/upload_images/13714448-8a388dfa095ffe4c.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 直接运行本单元格即可，本单元格代码的作用是：忽略下面代码执行过程中的版本警告等无用提示\n",
    "\n",
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "词袋方法进行文本特征工程，使用sklearn默认的逻辑回归分类器，验证集上的预测准确率: 0.6430603613994618\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lg1 = LogisticRegression()\n",
    "lg1.fit(x_train,y_train)\n",
    "print('词袋方法进行文本特征工程，使用sklearn默认的逻辑回归分类器，验证集上的预测准确率:',lg1.score(x_test,y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多项式朴素贝叶斯分类器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "词袋方法进行文本特征工程，使用sklearn默认的多项式朴素贝叶斯分类器，验证集上的预测准确率: 0.6084070229398949\n"
     ]
    }
   ],
   "source": [
    "#引用朴素贝叶斯进行分类训练和预测\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(x_train,y_train)\n",
    "print('词袋方法进行文本特征工程，使用sklearn默认的多项式朴素贝叶斯分类器，验证集上的预测准确率:',classifier.score(x_test,y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "多项式朴素贝叶斯分类器，训练速度很快，但准确率较低。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用TF-IDF模型进行文本特征工程\n",
    "\n",
    "TF值衡量了一个词出现的次数。<br>\n",
    "IDF值衡量了这个词是不是烂大街。如果是the、an、a等烂大街的词，IDF值就会很低。<br>\n",
    "两个值的乘积TF_IDF反映了一个词的出现带来的特异性信息。<br>\n",
    "例如，“中国”、“功夫”这两个词也许会同时出现，但“中国”这个词在各个文档中都有出现，IDF值很低，因此TF_IDF值也很低。\n",
    "而“功夫”这个词只在特定文档中出现，这个词能带来的“特异性”信息就会大很多。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{aligned}\n",
    "TF(“功夫\")&=\\frac{\"功夫\"这个词在当前文章中出现的次数}{\"功夫\"这个词在整个语料库中出现的次数}\\\\\n",
    "\\\\\n",
    "\\\\\n",
    "IDF(“功夫\")&=ln \\frac{语料库的总文档数}{语料库中\"功夫\"出现的文档数}\\\\\n",
    "\\\\\n",
    "\\\\\n",
    "TF\\_IDF(“功夫\")&=IF(“功夫\") \\times IDF(“功夫\")\n",
    "\\end{aligned}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 用sklearn库中的TfidfVectorizer构建TF-IDF模型\n",
    "# TF-IDF模型的详细介绍请看子豪兄的视频\n",
    "# analyzer='word'指的是以词为单位进行分析，对于拉丁语系语言，有时需要以字母'character'为单位进行分析\n",
    "# ngram指分析相邻的几个词，避免原始的词袋模型中词序丢失的问题\n",
    "# max_features指最终的词袋矩阵里面包含语料库中出现次数最多的多少个词\n",
    "\n",
    "# TF-IDF模型是专门用来过滤掉烂大街的词的，所以不需要引入停用词stop_words\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tf = TfidfVectorizer(\n",
    "    analyzer='word',\n",
    "    ngram_range=(1,4),\n",
    "    # stop_words=stop_words,\n",
    "    max_features=150000\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.float64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=150000, min_df=1,\n",
       "        ngram_range=(1, 4), norm='l2', preprocessor=None, smooth_idf=True,\n",
       "        stop_words=None, strip_accents=None, sublinear_tf=False,\n",
       "        token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b', tokenizer=None, use_idf=True,\n",
       "        vocabulary=None)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.fit(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "类似上面的操作，拆分原始训练集为训练集和验证集，用TF-IDF模型对每一个词都进行特征工程，变成向量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "x_train,x_test,y_train,y_test = train_test_split(train_sentences,label,random_state=1234)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = tf.transform(x_train)\n",
    "x_test = tf.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x150000 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 14 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 构建分类器算法，对TF-IDF模型处理后的文本进行机器学习和数据挖掘"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 朴素贝叶斯分类器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TF-IDF方法进行文本特征工程，使用sklearn默认的多项式朴素贝叶斯分类器，验证集上的预测准确率: 0.6045367166474432\n"
     ]
    }
   ],
   "source": [
    "#引用朴素贝叶斯进行分类训练和预测\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(x_train,y_train)\n",
    "print('TF-IDF方法进行文本特征工程，使用sklearn默认的多项式朴素贝叶斯分类器，验证集上的预测准确率:',classifier.score(x_test,y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 逻辑回归分类器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TF-IDF方法进行文本特征工程，使用sklearn默认的逻辑回归模型，验证集上的预测准确率: 0.6326541073945918\n"
     ]
    }
   ],
   "source": [
    "# sklearn默认的逻辑回归模型\n",
    "lg1 = LogisticRegression()\n",
    "lg1.fit(x_train,y_train)\n",
    "print('TF-IDF方法进行文本特征工程，使用sklearn默认的逻辑回归模型，验证集上的预测准确率:',lg1.score(x_test,y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TF-IDF方法进行文本特征工程，使用增加了两个参数的逻辑回归模型，验证集上的预测准确率: 0.6533384595668332\n"
     ]
    }
   ],
   "source": [
    "# C：正则化系数，C越小，正则化效果越强\n",
    "# dual：求解原问题的对偶问题\n",
    "lg2 = LogisticRegression(C=3, dual=True)\n",
    "lg2.fit(x_train,y_train)\n",
    "print('TF-IDF方法进行文本特征工程，使用增加了两个参数的逻辑回归模型，验证集上的预测准确率:',lg2.score(x_test,y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对比两个预测准确率可以看出，在逻辑回归中增加C和dual这两个参数可以提高验证集上的预测准确率，但如果每次都手动修改就太麻烦了。我们可以用sklearn提供的强大的网格搜索功能进行超参数的批量试验。<br>\n",
    "搜索空间：C从1到9。对每一个C，都分别尝试dual为True和False的两种参数。<br>最后从所有参数中挑出能够使模型在验证集上预测准确率最高的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise-deprecating',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
       "          n_jobs=None, penalty='l2', random_state=None, solver='warn',\n",
       "          tol=0.0001, verbose=0, warm_start=False),\n",
       "       fit_params=None, iid='warn', n_jobs=-1,\n",
       "       param_grid={'C': range(1, 10), 'dual': [True, False]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "param_grid = {'C':range(1,10),\n",
    "             'dual':[True,False]\n",
    "              }\n",
    "lgGS = LogisticRegression()\n",
    "grid = GridSearchCV(lgGS, param_grid=param_grid,cv=3,n_jobs=-1)\n",
    "grid.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 5, 'dual': True}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后超参数搜索的结果是，C为5，dual为True，能够使逻辑回归模型在验证集上预测准确率最高。我们便采用这个最优参数，构建lg_final分类器，最终在验证集上预测正确率为0.655464。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "lg_final = grid.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "经过网格搜索，找到最优超参数组合对应的逻辑回归模型，在验证集上的预测准确率: 0.6546456491093169\n"
     ]
    }
   ],
   "source": [
    "print('经过网格搜索，找到最优超参数组合对应的逻辑回归模型，在验证集上的预测准确率:',lg_final.score(x_test,y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 对测试集的数据进行预测，提交Kaggle竞赛最终结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>SentenceId</th>\n",
       "      <th>Phrase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>156061</td>\n",
       "      <td>8545</td>\n",
       "      <td>An intermittently pleasing but mostly routine ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>156062</td>\n",
       "      <td>8545</td>\n",
       "      <td>An intermittently pleasing but mostly routine ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>156063</td>\n",
       "      <td>8545</td>\n",
       "      <td>An</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>156064</td>\n",
       "      <td>8545</td>\n",
       "      <td>intermittently pleasing but mostly routine effort</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>156065</td>\n",
       "      <td>8545</td>\n",
       "      <td>intermittently pleasing but mostly routine</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PhraseId  SentenceId                                             Phrase\n",
       "0    156061        8545  An intermittently pleasing but mostly routine ...\n",
       "1    156062        8545  An intermittently pleasing but mostly routine ...\n",
       "2    156063        8545                                                 An\n",
       "3    156064        8545  intermittently pleasing but mostly routine effort\n",
       "4    156065        8545         intermittently pleasing but mostly routine"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看测试集数据前5行，Phrase列就是需要我们自己构建模型预测情感标签的文本\n",
    "data_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用TF-IDF对测试集中的文本进行特征工程\n",
    "test_X = tf.transform(data_test['Phrase'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对测试集中的文本，使用lg_final逻辑回归分类器进行预测\n",
    "predictions = lg_final.predict(test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 2, 3, ..., 1, 1, 2], dtype=int64)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(66292,)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将预测结果加在测试集中\n",
    "\n",
    "data_test.loc[:,'Sentiment'] = predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>SentenceId</th>\n",
       "      <th>Phrase</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>156061</td>\n",
       "      <td>8545</td>\n",
       "      <td>An intermittently pleasing but mostly routine ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>156062</td>\n",
       "      <td>8545</td>\n",
       "      <td>An intermittently pleasing but mostly routine ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>156063</td>\n",
       "      <td>8545</td>\n",
       "      <td>An</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>156064</td>\n",
       "      <td>8545</td>\n",
       "      <td>intermittently pleasing but mostly routine effort</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>156065</td>\n",
       "      <td>8545</td>\n",
       "      <td>intermittently pleasing but mostly routine</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PhraseId  SentenceId                                             Phrase  \\\n",
       "0    156061        8545  An intermittently pleasing but mostly routine ...   \n",
       "1    156062        8545  An intermittently pleasing but mostly routine ...   \n",
       "2    156063        8545                                                 An   \n",
       "3    156064        8545  intermittently pleasing but mostly routine effort   \n",
       "4    156065        8545         intermittently pleasing but mostly routine   \n",
       "\n",
       "   Sentiment  \n",
       "0          2  \n",
       "1          2  \n",
       "2          3  \n",
       "3          2  \n",
       "4          2  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 按Kaggle比赛官网上的要求整理成这样的格式\n",
    "\n",
    "final_data = data_test.loc[:,['PhraseId','Sentiment']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>156061</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>156062</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>156063</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>156064</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>156065</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PhraseId  Sentiment\n",
       "0    156061          2\n",
       "1    156062          2\n",
       "2    156063          3\n",
       "3    156064          2\n",
       "4    156065          2"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 保存为.csv文件，即为最终结果\n",
    "\n",
    "final_data.to_csv('final_data.csv',index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 扩展阅读\n",
    "\n",
    "除了词袋模型和TF-IDF模型之外，还有其它文本数据特征工程的方法，比如word2vec。<br>\n",
    "你可以在这个网站http://projector.tensorflow.org/ \n",
    "看到word2vec的可视化效果，虽然这个效果是降到三维之后的，但仍然能看到关联词之间的关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 微信扫描二维码给我打赏哦，支持我们做出更多的人工智能和机器学习视频教程。\n",
    "\n",
    ">Bilibili视频专栏：同济子豪兄"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![子豪兄赞赏码.png](https://upload-images.jianshu.io/upload_images/13714448-7be6682a94b8211b.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
